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Abstract 

Pilot aiding during low-altitude flight depends on 
the ability to detect and locate obstacles near the 
helicopter’s intended flight path. Computer-vision- 
based methods provide one general approach for 
obstacle detection and range estimation. Several 
algorithms have been developed for this purpose, 
but have not been tested with actual flight data. 
This paper presents results obtained using helicopter 
flight data with a feature-based range estimation al- 
gorithm. A method for recursively estimating range 
using a Kalman filter with a monocular sequence of 
images and knowledge of the camera’s motion is de- 
scribed. The helicopter flight experiment and four 
resulting datasets are discussed. Finally the perfor- 
mance of the range estimation algorithm is explored 
in detail based on comparison of the range estimates 
with true range measurements collected during the 
flight experiment. 


1 Introduction 

NASA Ames in conjunction with the U.S. Army has 
been investigating the use of passive range estima- 
tion with an imaging sensor to detect and locate 
obstacles which may represent a safety hazard to he- 
licopters during low-altitude flight [1]. The obstacle 
information may be displayed directly to the pilot or 
used by an automatic obstacle-avoidance guidance 
system to enhance safety for near-terrain flight. 

Several approaches for implementing passive 
range estimation have been investigated at NASA 
Ames [2, 3, 4, 5]. In previous works, simulated and 
laboratory data have been used to demonstrate the 
performance of these algorithms. More recently, a 
helicopter flight experiment has been conducted to 
obtain samples of data that will be available from 
onboard systems during flight. The results of one 


passive range estimation algorithm using the heli- 
copter data is presented in this paper to demonstrate 
the performance that can be expected in flight. 

The range estimation algorithm presented here 
is based on a monocular sequence of images and on 
knowledge of the camera’s motion. Optic flow mea- 
surements are derived by a feature-based method 
which tracks distinctive regions between images. An 
extended Kalman filter uses the measurements of op- 
tic flow and camera motion to recursively estimate 
range. The recursive nature of the Kalman filter pro- 
vides an efficient mechanism for combining redun- 
dant measurements to successively refine the range 
estimates. 

The helicopter flight experiment provides a 
realistic flight database including video images, 
motion state information, and camera calibration 
parameters. True range measurements were also 
obtained for evaluating the accuracy of range esti- 
mates. Straight-line and curved flight paths were 
flown over man-made and natural terrain. 

The purpose of this paper is to present range 
estimation results using actual flight data, and to 
evaluate algorithm performance by comparison of 
range estimates with true range measurements. The 
range estimation algorithm is described in Section 2, 
focusing on the extended Kalman filter. Section 3 
describes the helicopter flight experiment and the 
contents of the resulting flight database. Range es- 
timation results obtained from four flight scenarios 
are the subject of Section 4. Conclusions and ideas 
for future work complete the paper. 


2 Estimation Algorithm 

Consider a helicopter-mounted camera that observes 
a point P on a stationary object in the environment 
as shown in Figure 1. The image point {u,v} corre- 
sponding to the point P is given by the perspective 


1 



Figure 1: Geometry for Passive Range Estimation 

projection equations as follows 

u = fx 3 /z 8y v = fy 9 /z 8 (1) 

where x 3 , y 8 , z* are components of p 3y the object’s 
position relative to the camera, and / is the focal 
length of the camera lens. As the camera moves, 
the image of P will also move. If P is assumed fixed 
in the Earth frame, the rate of change of p 8 in the 
camera’s axes system can be determined using the 
Coriolis equation as follows 

Pb = ~V 8 u> 3 x p 8 (2) 

where V 8 = {lx*, Vz*} and u? 3 = 
are the camera’s translational and rotational veloci- 
ties with respect to the Earth frame. Differentiating 
the perspective projection equations with respect to 
time and substituting for p 8 according to the above 
equation yields the well-known optic flow equations 
which relate camera motion, object motion in the 
image, and the object’s range. The optic flow equa- 
tions are as follows 

u = (—fV X8 + vV z8 )/z 3 

- f + js ) v y3 + vu Z3 

V - (-fV ys + vV Z8 ) /z 3 

+/ (i + jfc) ^9 “ *fu y3 - UV X8 . (3) 

Decomposing the optic flow into components due to 
camera translation t> t ) and rotation (ur , v r ) gives 
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With the optic flow equations, range to an object z s 
can be determined given measurements of the cam- 
era’s motion state (V 3y uj 3 ) y the object’s location in 
the image (u, t/), and the optic flow (ft, v). Note that 
only the optic flow due to translation {u ty v t ) is a 
function of the object’s range; therefore, range can 
be estimated only when the camera is translating. 
Furthermore, the optic flow due to translation will 
be zero at an image location (fV X3 /V Z8y fV y8 /V Z3 ) 
known as the focus of expansion (FOE). Since the 
FOE corresponds to the intersection of the velocity 
vector with the image plane, range estimates cannot 
be obtained for objects along the camera’s instan- 
taneous direction of motion. Additionally, the abil- 
ity to estimate range deteriorates for objects whose 
translational optic flow is small, such as objects that 
are far away or that appear near the FOE. 

Optic-flow information and ultimately a range 
estimate is associated with features in an image, 
where a feature is defined as a small region of in- 
terest within the image. The optic flow measure- 
ments are obtained from the difference between an 
object’s location in successive images. The number 
of features for which range estimates can be obtained 
depends directly on the ability to select robust fea- 
tures that can be unambiguously tracked between 
images. The quality of the range estimate for each 
feature depends on the ability to accurately track 
the feature between images. In this implementation 
features are squares of 11 x 11 pixels which exhibit 
intensity variation greater than some user-specified 
minimum threshhold value. A correlation method is 
used to determine the feature’s location in each new 
image. 

Using the optic-flow equations (3), we formu- 
late the computation of range as a state estimation 
problem using a Kalman filter. The Kalman filter is 
well-suited to this application because it combines 
redundant measurements to recursively improve its 
estimate over time. In addition, the state covari- 
ance matrix provided by the Kalman filter gives an 
indication of the estimate accuracy. The Kalman fil- 
ter also yields a prediction of the state vector, the 
state covariance matrix, and an expected location 
of the feature for the next sample time. This latter 
information is used to constrain the search area for 
locating the feature in the next image. As the range 
information improves, the search window becomes 
smaller and less computation is required to locate 
the feature. 

Several Kalman filter implementations were 
studied by Sridhar and Phatak [2], who obtained 
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the best results by selecting the state vector X = p s 
and the measurement vector Z = With these 

definitions, the Coriolis equation (2) becomes the 
state equation and the perspective projection equa- 
tions (1) become the measurement equations. The 
state and measurement equations can be written as 
follows 


x = -Wi-K 

Z = h(X) = [fx s /z s ,fy s /z s ] T 

where 

0 -U xs 
aJ, xs 0 
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The state equation is a time-varying hnear system 
that depends on the camera’s translational and ro- 
tational velocities. The measurement equation is a 
nonlinear function of the state. 

The continuous- time state and measurement 
equations can be converted to their discrete- time 
equivalents assuming that V a and are constant 
during the sampling interval AT. The discrete time 
system equations are 


~ X(k + 1) = $(k)X(k) + T(k)U(k) 

+r d (A 0Cx(&) (8) 

Z(k) = h{X(k)] + Uk) (9) 

where $(fc) is the state transition matrix, T(k) is 
the input distribution matrix, U(k) = -V s {k) is the 
control matrix, Td{k) is the disturbance distribu- 
tion matrix, and C x (fc) and Cz(^) model the process 
noise and measurement noise, respectively. Zero- 
mean Gaussian white noise is assumed such that 
R(k) = cov((lz) and Q(k) = cov (C x ). The state tran- 
sition matrix and the control distribution matrices 
have been derived by Sridhar and Phatak [2]. The 
measurement equation is linearized about the cur- 
rent estimate of X giving 


Z(k) 

H(k) 


H(k)X(k) + C*(k) 

dh(X)/dX 

f \ !/*. 0 -*#/•*? ’ 

J [ 0 l/z. -y./z* 


( 10 ) 

( 11 ) 

P*=P. (k- 1) 


where H(k) is computed based upon the best state 
estimate available just before the measurement up- 
date. The discrete-time state equation (8) and the 
linearized measurement equation (10) are used to re- 
cursively estimate the state vector X and the state 
covariance matrix P. 


The Kalman filter consists of two parts: the 
measurement update which improves the state esti- 
mate given a new measurement, and the time update 
which propagates the state forward in time accord- 
ing the the system dynamics. Before each iteration 
of the Kalman filter, we know Q(k) and R(k) and we 
have estimates of X(k) and P(k). The measurement 
update is then performed according to the following 
equations 

X(k) = X(k) + K(k)[Z(k) - H(k)X(k)] 

P(k) = [I - K(k)H(k)\P(k) (12) 

where H(k) is computed from X(k) as described 
above and the Kalman filter gain K(k) is computed 
using the equation 

K(k) = P(k)H T (k)[H(k)P(k)H T (k ) + R(k)]~ l . 

. (13) 

The time update equations are 

A' (k + 1 ) = $(k)X(k) + r(k)U(k) 

P(k + 1) = $(k)P(k)$(k) T 

+T d (k)Q(k)T d (k) T (14) 

As noted above, the Kalman filter requires ini- 
tial estimates for X and P. The initial estimate 
for X can be derived from the optic flow equations 
and the perspective projection equations given a fea- 
ture’s location in two images and the camera’s trans- 
lational and rotational velocities which are assumed 
constant during the interw.l between images. First, 
the optic flow equations (3) are solved for z s . The 
optic-flow equations actually comprise an over deter- 
mined system of two equations in the one unknown 
z 8 , so a single quadratic equation in z s is formed by 
summing the squares of the two optic-flow equations. 
Once z s is found, x 8 and y 6 can be determined from 
the perspective projection equations (1). The initial 
estimate of the state covariance matrix is chosen a 
priori . 

A feature belonging to a far-away object or a 
feature near the FOE may have an inter-image mo- 
tion smaller than can be resolved by the measure- 
ment process. The effective signal-to-noise ratio of 
shift measurements can be increased by skipping im- 
age frames to lengthen the time interval between im- 
ages. Feature matching will then take place between 
images that are n frames apart, where n is the frame 
skipping factor. The Kalman-filter time updating 
is still performed at the original rate because the 
camera’s motion information does not change. The 
measurement update is performed less frequently be- 
cause each feature’s location is measured only every 
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Figure 2: Flight Experiment Overview 

n frames. When the time index A; is a multiple of 
n, a full measurement update is performed accord- 
ing to equations (12,13) above; otherwise, a trivial 
measurement update is performed according to the 
following equations 

X(k) = X(k) 
p(k) = p(k) 

Note that for a frame skipping factor of one every 
image is processed and the modified measurement 
update equations degenerate to the original form 
given in equations (12,13). The effect of varying 
measurement update rate will be discussed in Sec- 
tion 4 where a reference image sequence will be sub- 
sampled using several values for the frame skipping 
factor. 

3 Experimental Flight Data 

In order to test the range estimation algorithm with 
actual data, a helicopter flight experiment was con- 
ducted at NASA Ames [6, 7]. The measurements 
included video imagery, the helicopter motion state, 
true range information, and the camera’s character- 
istic parameters. The flight experiment isillustrated 
in Figure 2. 

A monochrome video camera was mounted in 
the nose of a CH-47 Chinook helicopter for image 
collection. The video signal was recorded onboard 
the helicopter with a U-matic video recorder and 
later digitized for computer processing. State mea- 
surements were acquired from an inertial navigation 
unit (INU) and discrete sensors such as accelerom- 
eters and rate gyros onboard the helicopter. The 
true range information was obtained using a ground- 
based laser tracker to measure the position of the 
helicopter-mounted camera and the (stationary) tar- 
gets. Correlation between the airborne data (video 


imagery and state data) and the ground laser posi- 
tion data was accomplished using synchronized time 
sources. On the helicopter a time message was su- 
perimposed in a corner of the image and on the 
ground the measurements were stamped with the 
current time. 

Knowledge of the camera’s installation geome- 
try and its imaging properties is also necessary to 
perform range estimation. It was assumed in the al- 
gorithm development that the camera’s motion was 
available in the camera’s axis system, and that an 
image location could be directly measured in terms 
of u and v as given in the perspective projection 
equations. In each case, the desired information 
must be derived from raw measurements and knowl- 
edge of the camera calibration parameters. The 
camera parameters consist of external parameters 
which describe the camera’s geometric installation 
including its position and orientation with respect 
to the helicopter’s axis system, and internal param- 
eters which describe the imaging properties of the 
camera such as the lens focal length, the spacing of 
image pixels, and the pixel location where the z B axis 
intersects the image plane. The camera parameters 
were determined experimentally as described in an 
earlier paper [6]. 

During post-flight processing, the raw state 
measurements were filtered and checked for consis- 
tency. The measurements were sampled at 100 Hz 
and have a signal bandwidth of 10 Hz. Measure- 
ments of velocity, although available on most oper- 
ational helicopters, could not be obtained with the 
available equipment on the CH-47 and had to be de- 
rived from the remaining state measurements (posi- 
tion, acceleration, orientation, and rotational veloc- 
ity) using a state-estimation technique. Instrument 
bias and scale-factor errors were also identified and 
removed using state estimation. The resulting state 
information was converted to the camera’s axis sys- 
tem and subsampled to video rate (30 Hz). The 
variables which describe the relation between the 
helicopter body axes and the camera axes are the 
external camera calibration parameters. The laser 
measurements of position were also corrected for the 
offset between the laser-track point and the camera’s 
location. The difference between the target loca- 
tions and the camera location gives the true range 
measurements (or more accurately, the true relative 
displacement vectors) which can be expressed in 
Earth, body, or camera axes. 

The analog video signal recorded onboard the 
helicopter was digitized to produce 512 x 512 pixel 
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images with 256 gray levels. Given the digitized im- 
ages, an image plane location can be measured in 
terms of its row and column indices in the pixel ar- 
ray. An image plane location in (u, v) coordinates 
must be derived using the following equations 

u = (n u - n Uo )6u c 
v = (n v - n Vo )6v e (15) 

where is the grid location of the pixel, 

(Hu 0 > 7 i Vo ) is the grid location where the z s axis 
passes through the image plane, and (6u e , 6v e ) is the 
effective pixel size. The variables n Uo , n Vo , 6u e , 
and the lens focal length / are the internal camera 
calibration parameters. 

The digitized images, camera state information, 
true range measurements, and camera calibration 
parameters resulting from a particular flight sce- 
nario have been collected into data sets consisting 
of a sequence of images with headers and a separate 
file containing the true target position data. The 
data sets for four flight scenarios have been collected 
into a database which may be requested from NASA 
Ames [7]. The four flight scenarios are described be- 
low: 

1. Line Data Set: The helicopter is flying a 

straight line path between two rows of trucks 
parked on a runway in this scenario. The he- 
licopter has an altitude of about 15 ft above 
the runway and is flying at a ground speed of 
35 ft/sec. At this speed, the helicopter moves 
roughly 1 ft between successive images. The 
image sequence consists of 240 frames (or 8 sec- 
onds of flight) during which time the helicopter 
travels approximately 290 ft. The location of 
two points on each truck has been measured to 
provide true range information. Range to the 
trucks varies between 200 and 800 ft relative to 
the camera’s initial position. 

2. Arc Data Set: In this scenario the helicopter 
follows an S-shape ground track between the 
same arrangement of trucks described above. 
The helicopter alternates between flying toward 
the trucks on its left and those on its right. The 
helicopter maintains an altitude of 15 ft and a 
speed of 40 ft/sec. The peak yaw rate is approx- 
imately 0.13 rad/sec. During 8 seconds of flight 
(corresponding to a 240-frame image sequence) 
the helicopter moves 350 ft toward the trucks. 
The visible trucks are 200 to 650 ft away from 
the camera. 


3. Posts Data Set: This scenario shows the heli- 
copter flying a straight line path 35 ft above 
a field containing a road, fence posts, and a 
building with a tower. The helicopter’s speed 
is 40 ft/sec. The image sequence contains 
90 frames (3 seconds of flight) and the cam- 
era’s travel is 120 ft. The true range has been 
measured for 13 objects varying between 80 and 
350 ft from the camera’s initial position. 

4. Towers Data Set: The helicopter flies toward 
several large power transmission towers against 
a background of hills while following a straight 
line flight path. The helicopter’s speed is 
90 ft /sec and the image sequence consists of 
90 frames. True range data are not available 
for this image sequence. 

The next section will present range results ob- 
tained with these four data sets using the algorithm 
discussed earlier. 


4 Range Estimation Results 

4.1 Evaluation Method 

It is difficult to define a single measure for evaluat- 
ing range estimation algorithms. Potentially, range 
accuracy can be evaluated on a pixel-by-pixel basis 
provided that sufficient truth data are available. In 
practice range estimates and truth data are sparse 
so that overall algorithm performance depends not 
only on range accuracy but also on the distribution 
of image features where range estimates are avail- 
able. Because the distribution of image features re- 
sulting from different algorithms will vary, a more 
meaningful comparison is based on groups of fea- 
tures belonging to a single object. The resulting ob- 
ject range and size information is precisely the input 
required to plan an obstacle-free flight path. For this 
reason, the results presented here focus on compar- 
ison of (a) the range estimates of various objects, 
and (b) the range estimates of the individual fea- 
tures belonging to a single object. It is important to 
note that the algorithm described earlier automati- 
cally selects features and generates range estimates. 
Only when this process is complete are the resulting 
range estimates manually grouped for the purpose 
of performance evaluation. 
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Figure 3: First and Last Images of the Line Sequence 


The four data sets described in the previ- 
ous section have been chosen to demonstrate ma- 
neuvers and/or obstacles encountered during low- 
altitude helicopter flight. The Line sequence demon- 
strates the algorithm’s performance during nomi- 
nally straight and level flight conditions. The addi- 
tive effect of curvilinear motion is provided in the 
Arc sequence. The Posts and Towers sequences 
contain obstacles likely to be encountered in low- 
altitude flight. The results obtained with the Line 
sequence are described at length in the next section. 
The remaining sequences have been studied in simi- 
lar detail, but for brevity only the unique aspects of 
each will be discussed. 

4.2 Line Data Set 

The first and last images of the 240-frame Line se- 
quence taken during straight line flight are shown in 
Figure 3. The labeled points in the first image indi- 
cate locations where the true range has been mea- 
sured. The camera’s path and the measured truck 
locations are shown in Figure 4. The camera’s mea- 
sured velocity is shown in Figure 5. The helicopter’s 
attitude changes continuously even during straight 
and level flight which manifests itself as consider- 
able motion of the FOE. Using the measurements of 
V s and u s , the FOE location has a range of travel of 
about 40 pixels both horizontally and vertically as 
shown in Figure 6. 
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Figure 4: Camera Path and Truck Locations for Line 
Sequence 


Range estimation was performed on this se- 
quence with the parameters 


P(0) « 
R(k) = 
Q(k ) = 
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Figure 7 shows the image locations where range es- 
timates were obtained in the 65th frame of the se- 
quence. The FOE’s location for this image is de- 
picted by crosshairs in the figure. A total of ap- 
proximately 250 range estimates were obtained in 
the image shown. The range estimates belonging to 
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Figure 5: Camera Velocity for Line Sequence 



Figure 7: Range Estimates for Line Sequence 

(frame 65) 



Figure 6: FOE Motion for Line Sequence 


each truck are indicated by the boxes in Figure 7. 
The range estimates for truck A are plotted in Fig- 
ure 8 along with the true range to the front and 
rear of the truck. Some of the range estimates start 
to die out near frame 70 as the truck begins mov- 
ing out of the camera’s field of view. The initial 
range estimates for new features may contain large 
errors because the optic flow is small and because 
the initial range estimate is based on only two mea- 
surements. Truck A, for example, which is favorably 
located near the camera and far from the FOE, ha.s 
an inter-image shift of about 0.7 pixel relative to the 
FOE at the beginning of the sequence. In contrast . 
Truck E is both far from the camera and close to the 
FOE, leading to an initial inter-image shift of only 
0.1 pixel. During the same time, the FOE moves by 
approximately 1.5 pixels. The difficulty in measur- 
ing these small feature motions makes the process 
of range estimation for the flight data significantly 
more challenging and demonstrates the importance 
of subpixel measurement accuracy for this applica- 
tion. Figure 8 shows that even for poor initial range 
values, the Kalman filter will converge to a reason- 
able range estimate. The state covariance matrix 
provides an indication of the confidence associated 
with each estimate. The square root of the z s com- 
ponent of the state covariance matrix is shown in 
Figure 8 for the features belonging to truck A. 
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Figure 8: Range Estimates for Truck A in Line Sequence 


The mean and standard deviation of the se- 
lected range estimates belonging to a truck can be 
compared with the true range measurements for that 
truck in order to evaluate the performance of the 
range estimation algorithm. An improved estimate 
of the object’s range can be obtained by eliminating 
low-confidence range samples. Severed criteria were 
used to identify undesirable range estimates includ- 
ing the following: 

1. Percentage Method - Disregard the highest and 
lowest 20% of the range estimates. 

2. Statistical Method - Compute the raw mean 
and standard deviation of the range samples. 
Disregard these range estimates lying more 
than 2 standard deviations from the mean. 

3. Covariance Matrix Method - Use the state co- 
v ariance m atrix t o disregar d samples for which 
>/Pxs t z*(i) > ocy/P ZStX9 ( 0). Values of 10% and 
20% were used for a. 

The mean and standard deviations of the remain- 
ing samples give a composite range estimate for the 
truck. The resulting composite range estimates for 
truck A are shown in Figure 9. Each of the trim- 
ming methods leads to a composite range estimate 
which lies between the front and back of the truck. 
The range estimates begin to diverge from the true 
range at frame 70 because the truck begins mov- 
ing out of the camera’s field of view. The features 


tracking the front of the truck die off first leaving 
a greater proportion of range estimates to the back 
of the truck, so the mean of the remaining estimates 
slowly increases. The standard deviation of the com- 
posite range estimate decays over time in a roughly 
exponential manner. 

Similar range estimation results have been ob- 
tained for the remaining trucks as shown in Fig- 
ure 10. The range estimates (trimmed using the 
Percentage Method) and true range measurements 
are tabulated in Table 1 for every 2 seconds of flight. 
The true range column gives the range to the mid- 
point of each truck while the true standard deviar 
tion represents half of the truck’s length. As each 
truck is 20 feet in length, range errors less than 
about 10 feet are unimportant because they indi- 
cate range estimates which lie between the front and 
back of a truck. Table 1 shows that the range error 
and the standard deviation of the range estimate are 
greater for more-distant trucks. Both range errors 
and standard deviations decrease over time indicat- 
ing improved accuracy and a greater uniformity of 
the individual estimates. The range estimates con- 
verge to within 10% of the true range for each truck. 
The standard deviation is less than half the truck 
length for the nearest three trucks. 

The time required for the range estimation pro- 
cess to converge is also important in many applica- 
tions. Table 2 shows the number of frames required 
for the range estimates to converge to within 10% 
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Figure 10; Composite Range Estimates for Trucks B-E in Line Sequence 


Table 2: Convergence Time for Various Trimming Methods 


Thick 

Trimming Method 

Raw 

Percentage 

Statistical 

Cov (10%) 

Cov (20%) 


frame 

range 

frame 

range 

frame 

range 

frame 

range 

frame 

range 

A 

9 

269.9 

7 

271.9 

7 

271.9 

26 

252.7 

19 

259.9 

B 

65 

334.5 

24 

379.3 

24 

379.3 

45 

356.4 

32 

370.6 

C 

35 

493.7 

47 

481.1 

35 

493.7 

71 

455.3 

54 

473.7 

D 

94 

551.1 

125 

512.1 

123 

514.6 

168 

458.0 

141 

491.8 

E 

69 

705.9 

69 

705.9 

69 

705.9 

118 

648.1 

60 

715.6 
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of the true mean range as a function of the vari- 
ous trimming methods. The Percentage Method and 
Statistical Method yield the best results in terms of 
minimizing convergence time. The time until con- 
vergence generally increases with range and proxim- 
ity to the FOE. Truck D deviates from this trend, 
possibly due to tracking difficulties caused by dis- 
tracting background objects (see Figure 7). Table 2 
also gives each truck’s true range at the time when 
convergence is achieved. The minimum distance at 
convergence is about 250 ft. Since the helicopter 
moves at about 35 ft /sec, this would give a pilot 
a minimum of about 7 seconds of warning to avoid 
each of the trucks. In most cases, for a given distance 
of camera travel the algorithms have demonstrated 
the ability to estimate the range to objects up to 
10 times farther away to within 10% error. 

Inter-image feature shifts may be much less 
than a pixel as noted earlier. This is a difficulty 
since the ability to measure subpixel image motion 
is limited. The effective signal-to-noise ratio of shift 
measurements can be increased by skipping images 
of the original sequence. Range estimation was per- 
formed with skipping factors of 1, 2, 3, 4, 5, and 10. 
The resulting range estimates were trimmed using 
the Percentage Method and the time to 10% con- 
vergence was determined. The convergence times 
given in Table 3 are expressed as the equivalent 
frame number of the original sequence so that the 
results of different skipping factors can be compared 
on an equal footing. The columns of Table 3 show 
that for a given frame skipping factor, convergence 
time increases with range. The rows of Table 3 show 
that for each truck there is some optimal frame skip- 
ping factor, say n* , which yields a minimum conver- 
gence time (indicated by boxed entries in the table). 
Since Truck A exhibits little variation in convergence 
time, n* is chosen to be one. Truck E has two lo- 
cal minima, both indicated by boxed entries. For 
frame skipping factors less than n* the convergence 
time increases due to the limited accuracy of feature 
shift measurements, while for frame skipping factors 
greater than n* the convergence time increases due 
to the decreased measurement rate. Finally, Table 3 
shows the trend that the optimal frame skipping fac- 
tor for minimum convergence time increases with an 
object’s range. 


4.3 Arc Data Set 

The first and last images of the 240-frame Arc se- 
quence taken during maneuvering flight are shown in 


Table 3: Convergence Time for Various Frame Skip- 
ping Factors 
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Figure 11. The labeled points in the first image in- 
dicate locations where the true range has been mea- 
sured. The camera’s path and the measured truck 
locations are shown in Figure 12. The camera’s mea- 
sured linear and angular velocities are shown in Fig- 
ure 13. The FOE travels about 150 pixels horizon- 
tally and 40 pixels vertically as shown in Figure 14. 

The resulting range estimates (trimmed with 
the Percentage Method) are shown in Table 4. Truck 
D moves out of the field of view at frame 105 and 
re-enters the image at frame 190. The range esti- 
mate for Truck D (using the Percentage Method) is 
plotted in Figure 15. The range estimate for each 
truck is seen to converge to within about 10% of 
the true range. The number of frames required for 
convergence is given in Table 5 along with the true 
range when convergence is achieved. The minimum 
range at convergence is for Truck B at 229.5 ft which 
gives the pilot about 5.7 seconds of warning to react 
and plan a new path. Table 5 also gives the dis- 
tance traveled by the helicopter before convergence 
is reached and the percentage of the initial obsta- 
cle range represented by the travel distance. The 
distance traveled for Truck B is 13.7% of its initial 
range, roughly the same fraction as observed for the 
Line sequence. This is not unexpected because over 
the first 27 frames of the sequence when the range 
estimate is converging, the helicopter’s path is well 
approximated by a straight line (see Figure 12). The 
percentage distance traveled for Trucks C and D is 
probably a better indicator of the performance that 
can be expected during maneuvering flight. The 
convergence time for Truck E is significantly greater 
them for the other trucks although the range at con- 
vergence is comparable. The slower convergence for 
Truck E is attributed to its proximity to the FOE as 
the FOE sweeps across the image. Figure 16 shows 
the distance between the FOE and each of the two 
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Figure 11: First and Last Images of the Arc Sequence 
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Figure 12: Camera Path and Truck Locations for 
Arc Sequence 


known locations on Truck E as a function of the 
frame number. 

The convergence time of the range estimates 
during maneuvering flight is greater than for straight 
line flight. The Kalman-filter-based range estima- 
tion procedure is expected to perform equally well 
under both flight scenarios and does not provide an 
adequate explanation for the performance degrada- 
tion. More probably, the performance difference is 
due to errors in the motion data used by the Kalman 
filter. The helicopter’s heading angle has the low- 
est resolution (0.7 degree) of the angular measure- 
ments collected during flight test. Normally, the 
helicopter’s heading angle would not enter into the 
motion data used for range estimation if the linear 
velocity and angular velocity were directly measured 


in the helicopter’s body axis system or in the cam- 
era’s axis system. Recall, however, that linear veloc- 
ity could not be directly measured during flight and 
instead had to be reconstructed using state estima- 
tion. The resulting velocity profile depends most 
heavily on the available position and acceleration 
measurements. The helicopter’s heading angle plays 
an important role in transforming the position infor- 
mation measured in Earth axes into the helicopter’s 
body axis system. An error of 0.7 degree in the 
helicopter’s heading angle leads to an 8-pixel error 
in computing the FOE location. In practice, since 
the estimated velocity profile is a compromise be- 
tween position and acceleration information, FOE 
location errors less than 8 pixels are expected. For 
straight and level flight the heading angle is slowly 
changing and the heading angle measurement error 
can be identified as a pure bias based on consis- 
tency between the position and acceleration mea- 
surements. The resulting FOE location errors are 
therefore minimal for straight line flight. During 
maneuvering flight, however, the heading-angle mea- 
surement error is only partially accounted for by bias 
correction. The remaining heading-angle measure- 
ment error translates directly into uncertainty in 
the FOE location which introduces errors into the 
range estimation process. In future flight experi- 
ments, this difficulty will be corrected by obtaining 
direct linear-velocity measurements or by obtaining 
heading-angle measurements of higher accuracy. 
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Figure 13: Camera Velocity for Arc Sequence 
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Table 4: Errors in Range Estimates of Trucks in Arc Sequence 


Truck 

Frame 

B 

59 

C 

59 

c 

119 

c 

179 

D 

59 

D 

105 

D 

239 

E 

59 

E 

119 

E 

179 

E 

239 


True 

Range 


182.2 


313.8 

242.6 

158.1 


% Error 
Range I Range 


Table 5: Converge Time for Arc Sequence 


Truck 

Convergence 

Frame 

B 

27 

C 

80 

D 

92 

E 

239 


True Range at 
Convergence 


Initial 

Range 


265.8 

375.5 

514.2 

621.7 


Distance 

Traveled 


% of Initial 
Range Traveled 


13.7 
22.6 
26.9 

50.7 
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Table 6: Errors in Range Estimates of Objects in Posts Sequence 


Truck 

Frame 

true 

Range 

True 

Std 

Est. 

Mean 

Est. 

Std 

Error 

Range 

% Error 
Range 

A 

35 

129.80 

0.00 

137.20 

0.90 

7.40 

5.70 

B 

55 

130.60 

0.00 

134.80 

0.30 

4.20 

3.22 

C 

89 

137.10 

0.00 

142.80 

4.00 

5.70 

4.16 

D 

89 

194.10 

0.00 

192.30 

0.00 

1.80 

0.93 

E 

89 

301.30 

9.40 

302.70 

7.30 

1.40 

0.46 

F 

89 

304.30 

0.50 

313.50 

5.00 

9.20 

3.02 

G 

89 

401.50 

4.40 

354.20 

19.60 

47.30 

11.78 

H 

89 

192.10 

0.00 

268.80 

0.00 

76.70 

39.93 

I 

89 

129.50 

0.00 

130.50 

5.30 

1.00 

0.77 

J 

89 

87.40 

0.00 

92.80 

1.50 

5.40 

6.18 

K 

70 

99.40 

0.00 

103.50 

2.70 

4.10 

4.12 

L 

70 

97.50 

0.00 

95.30 

2.40 

2.20 

2.26 

M 

55 

93.30 

0.00 

91.50 

2.00 

1.80 

1.93 




Figure 14: FOE Motion for Arc Sequence 


Figure 15: Composite Range Estimate for Truck D 
in Arc Sequence 




Frame Number 


Figure 16: Proximity of Truck E to the FOE for Arc 
Sequence 


4.4 Posts and Towers Data Sets 

The first and last images of the 90-frame Posts se- 
quence taken during 3 seconds of straight-line flight 
over a field with a road, a building, and several white 
posts are shown in Figure 17. The labeled points 
in the first image indicate locations where the true 
range has been measured. The sensor velocity is 
approximately 40 ft/sec during the sequence. The 
range estimates and true range measurements are 
shown in Table 6 for each of the objects labeled in 
Figure 17. The range estimate for an object corre- 
sponds to the last image frame in which the object 
appears. The range estimates are within 10% of the 
true value for all of the objects except G and H. 
Due to its proximity to the FOE, the range estimate 
for object G requires greater time to converge. Post 
H was first detected in frame 84 and requires more 
than the remaining 5 frames to converge. 

Figure 18 shows the first and last images of the 

90-frame Towers sequence taken during 3 seconds 
of straight-line flight over power transmission tow- 
ers. The sensor velocity is approximately 90 ft /sec 
during the sequence. True range measurements are 
not available for this sequence, so the range esti- 
mates can only be compared for consistency among 
the three towers indicated in Figure 18. The range 
estimates are 456.0, 442.0, and 441.1 feet for towers 
A, B, and C, respectively. The standard deviation 
of the estimates are 17.7, 15.4, and 15.6 feet. 
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5 Conclusions 


In this paper actual flight data have been used 
to demonstrate the feasibility of passive range es- 
timation from a helicopter equipped with a single 
camera and an inertial navigation system. Range 
estimation results were compared with true range 
measurements which were independently acquired. 
A feature-based method extracts optic-flow mea- 
surements from the image sequence. Then an ex- 
tended Kalman filter uses the measurements of op- 
tic flow and camera motion to recursively estimate 
range. Range estimation results have been shown 
for straight-line and maneuvering flight paths over 
man-made and natural terrain. 

The range estimation method yields results 
which compare closely with true range measure- 
ments. For straight-line motion, range estimates 
converging to within 10% of the true range by the 
time the helicopter has traversed l/10th the distance 
to the object. For straight-line flight at 35 ft /sec, ob- 
stacle ranges were estimated within 10% error at a 
minimum distance of 250 ft, giving the pilot approx- 
imately 7 seconds of warning to avoid the obstacles. 
During maneuvering flight the range estimates con- 
verged more slowly due to the relatively low reso- 
lution of angular motion information. For tu rnin g 
maneuvers at 40 ft /sec, obstacle ranges were esti- 
mated within 10% error at a minimum distance of 
230 ft, giving the pilot approximately 5.7 seconds of 
warning to avoid the obstacles. 

The inter-image motion of distant objects en- 
countered in the flight data was often significantly 
less than one pixel. For these objects, improved 
range accuracy and decreased convergence time are 
obtained by skipping images to effectively decrease 
the image sampling rate. In general, a particular 
value of the frame skipping factor leads to a mini- 
mum convergence time and this optimal frame skip- 
ping factor is seen to increase with the obstacle’s 
range. An event-driven Kalman filter is currently 
being explored as one method for simultaneously 
treating quickly and slowly moving features. Here 
the Kalman filter time update occurs with every im- 
age as before, but the measurement update takes 
place only when the feature has moved some min- 
imum distance (say, one pixel) from its previous 
measurement-update location. 

The motion-based range estimation algorithm 
described here suffers difficulties near the FOE as 
do most motion-based methods. An event-driven 
Kalman filter may help to minimize this problem 


by improving range estimates to slowly-moving fea- 
tures. In another approach, a combined stereo and 
motion range estimation algorithm has been devel- 
oped at Ames Research Center and tested with lab- 
oratory data [8]. A helicopter flight test program 
is under way to provide stereo images and motion 
measurements for use with this class of algorithms. 
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